The Hierarchy Consistency Index:

The attribute hierarchy method (AHM) (Leighton, Gierl, & Hunka, 2004), which is based on the assumption that test items can be described by a set of hierarchically ordered attributes, is designed to estimate examinees’ level of competency as well as their profiles for mastering a set of attributes. The AHM, by incorporating the assumption of attribute dependency, brings an important cognitive property into cognitive diagnostic testing. However, the validity of this new diagnostic model depends critically on the accuracy and adequacy of the attribute hierarchy. In this study, a person-fit statistic, called the hierarchy consistency index ( ), is introduced to help assess the degree to which an observed examinee response pattern is consistent with the attribute hierarchy of the AHM. This type of model-data fit study can enhance the validity of diagnostic feedback produced by the AHM. i HCI The Hierarchy Consistency Index 3 By estimating a person’s location on an underlying latent continuum, traditional assessments have been effective for selecting students who are most likely to succeed in a particular educational institution or program (Mislevy, 1995). Traditional assessments are typically constructed on logical taxonomies and content specifications but lack explicit cognitive models of the structures and cognitive processes that underlie student performance (Snow & Mandinach, 1991). As a result, test scores from traditional assessments are tied to content areas rather than the examinee’s cognitive processes measured by test items. In addition, test theories used for interpreting scores from traditional assessments are designed to optimize the estimate of a student’s single score on an underlying latent scale – the true score scale in classical test theory (CTT) or the latent trait scale in item response theory (IRT). A single aggregate score produced using CTT and IRT provides general information about students’ location on a continuum. However, it fails to provide specific information to inform teachers about their students’ cognitive strengths and weaknesses which may, in turn, help teachers make instructional decisions intended to help students succeed in educational settings (Nichols, 1994). Frustrated by the presence of these two limitations with traditional assessment approaches, measurement specialists have become increasingly interested in the development of new diagnostic assessments that are aimed at uncovering the cognitive processes used by students to respond to test items, determining the nature of poor performance, and classifying the poor performance in terms of an accepted typology of malfunctions (Scriven, 1999). As Nichols (1994) stated: These new assessments make explicit the test developer’s substantive assumptions regarding processes and knowledge structures a performer in a test domain would The Hierarchy Consistency Index 4 use, how the processes and knowledge structures develop, and how more competent performers differ from less competent performers. (p. 578) New diagnostic assessments enable researchers and educators to make inferences about cognitive processes and knowledge that students use when solving test items. A well-designed diagnostic assessment can measure different cognitive processes and knowledge required to solve test items in a domain of interest. Diagnostic assessments can also provide a profile of students’ mastery and non-mastery of cognitive skills. The value of diagnostic assessment lies in its ability to reveal each student’s specific cognitive strengths and weaknesses and further help design effective interventions for individual students. Cognitive diagnostic models (CDMs) have been developed to help construct diagnostic assessments and estimate students’ attribute mastery patterns associated with different cognitive skills. Leighton, Gierl, and Hunka (2004; see also Gierl, Leighton, & Hunka, 2000) proposed a CDM called the Attribute Hierarchy Method (AHM). The AHM is based on the assumption that test items can be described by a hierarchically-ordered set of attributes. Attributes are defined as basic cognitive processes or skills required to solve test items correctly (Leighton et al., 2004). The attribute hierarchy can be used as a basis for the development of test items, which upon the administration to examinees, produces vectors of binary responses (1 or 0). An examinee's response vector is then used to determine the examinee's level of competency as well as the likelihood that the attributes are possessed by the examinee. For the AHM to yield inferences about students, the attribute hierarchy in the domain of interest must be specified correctly. Hence, methods for assessing the accuracy and adequacy of the attribute hierarchy in describing the cognitive processes used by The Hierarchy Consistency Index 5 students to solve test items must be developed. Generally, methods for evaluating the misfit of an item-score vector to a specific test model have been referred to as "personfit" methods. Although numerous person-fit statistics have been proposed and investigated by researchers (Meijer & Sijtsma, 2001), as will be discussed shortly, most of these methods cannot be used to determine if the attribute hierarchy is truly representing the cognitive processes used by examinees to solve test items. Therefore, it is inappropriate to use these existing person-fit statistics with the AHM. The purpose of the present study is to introduce a person-fit statistic called the hierarchy consistency index ( ), which is designed explicitly to examine the degree to which an observed examinee response vector is consistent with the attribute hierarchy. The present paper is divided into five sections. The first section presents a brief overview of the AHM. The second section discusses the applicability of the existing person-fit statistics in the AHM framework and introduces the person-fit statistic, the , designed to assess whether the examinee i HCI i HCI uses different cognitive skills (or in a different combination) from what the attribute hierarchy indicates when solving test items. The third section introduces a simulation approach for setting the critical value of the . By comparing an observed against the critical value of the , researchers can determine whether the observed response pattern are consistent with the attribute hierarchy of the AHM. The fourth section applies the AHM and the to a real data set obtained from a previously administered large-scale achievement test for illustrative purposes. The fifth and final section summarizes and discusses two lines of future research related to the and its applicability to other cognitive diagnostic models. i HCI